Modern data systems can require the storage and communication of increasing amounts of data. Reducing the size of this data can reduce both the cost and time associated with storing and communicating the data.
In some circumstances, removing duplicate data can be used to reduce the size of the data, but this operation does not result in size reduction when the data does not have a lot of duplicates. Alternatively, data compression can reduce the size of non-duplicate data, but using this approach can cause deduplication to either not be performed, or performed with substantial overhead.